{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "efa6214c",
   "metadata": {
    "papermill": {
     "duration": 0.005588,
     "end_time": "2025-02-18T13:19:22.456297",
     "exception": false,
     "start_time": "2025-02-18T13:19:22.450709",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**This notebook is an exercise in the [Feature Engineering](https://www.kaggle.com/learn/feature-engineering) course.  You can reference the tutorial at [this link](https://www.kaggle.com/ryanholbrook/creating-features).**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8813c654",
   "metadata": {
    "papermill": {
     "duration": 0.004488,
     "end_time": "2025-02-18T13:19:22.465567",
     "exception": false,
     "start_time": "2025-02-18T13:19:22.461079",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction #\n",
    "\n",
    "In this exercise you'll start developing the features you identified in Exercise 2 as having the most potential. As you work through this exercise, you might take a moment to look at the data documentation again and consider whether the features we're creating make sense from a real-world perspective, and whether there are any useful combinations that stand out to you.\n",
    "\n",
    "Run this cell to set everything up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6a11d93a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:22.475794Z",
     "iopub.status.busy": "2025-02-18T13:19:22.475346Z",
     "iopub.status.idle": "2025-02-18T13:19:25.489868Z",
     "shell.execute_reply": "2025-02-18T13:19:25.488685Z"
    },
    "papermill": {
     "duration": 3.021851,
     "end_time": "2025-02-18T13:19:25.491897",
     "exception": false,
     "start_time": "2025-02-18T13:19:22.470046",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Setup feedback system\n",
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.feature_engineering_new.ex3 import *\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from xgboost import XGBRegressor\n",
    "\n",
    "\n",
    "def score_dataset(X, y, model=XGBRegressor()):\n",
    "    # Label encoding for categoricals\n",
    "    for colname in X.select_dtypes([\"category\", \"object\"]):\n",
    "        X[colname], _ = X[colname].factorize()\n",
    "    # Metric for Housing competition is RMSLE (Root Mean Squared Log Error)\n",
    "    score = cross_val_score(\n",
    "        model, X, y, cv=5, scoring=\"neg_mean_squared_log_error\",\n",
    "    )\n",
    "    score = -1 * score.mean()\n",
    "    score = np.sqrt(score)\n",
    "    return score\n",
    "\n",
    "\n",
    "# Prepare data\n",
    "df = pd.read_csv(\"../input/fe-course-data/ames.csv\")\n",
    "X = df.copy()\n",
    "y = X.pop(\"SalePrice\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b7a07f",
   "metadata": {
    "papermill": {
     "duration": 0.004416,
     "end_time": "2025-02-18T13:19:25.501488",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.497072",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "-------------------------------------------------------------------------------\n",
    "\n",
    "Let's start with a few mathematical combinations. We'll focus on features describing areas -- having the same units (square-feet) makes it easy to combine them in sensible ways. Since we're using XGBoost (a tree-based model), we'll focus on ratios and sums.\n",
    "\n",
    "# 1) Create Mathematical Transforms\n",
    "\n",
    "Create the following features:\n",
    "\n",
    "- `LivLotRatio`: the ratio of `GrLivArea` to `LotArea`\n",
    "- `Spaciousness`: the sum of `FirstFlrSF` and `SecondFlrSF` divided by `TotRmsAbvGrd`\n",
    "- `TotalOutsideSF`: the sum of `WoodDeckSF`, `OpenPorchSF`, `EnclosedPorch`, `Threeseasonporch`, and `ScreenPorch`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "32db5163",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.511794Z",
     "iopub.status.busy": "2025-02-18T13:19:25.511386Z",
     "iopub.status.idle": "2025-02-18T13:19:25.530040Z",
     "shell.execute_reply": "2025-02-18T13:19:25.528782Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.025729,
     "end_time": "2025-02-18T13:19:25.531725",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.505996",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_Q1\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "X_1 = pd.DataFrame()  # dataframe to hold new features\n",
    "\n",
    "X_1[\"LivLotRatio\"] = df.GrLivArea/df.LotArea\n",
    "X_1[\"Spaciousness\"] = (df.FirstFlrSF+df.SecondFlrSF)/df.TotRmsAbvGrd\n",
    "X_1[\"TotalOutsideSF\"] = df.WoodDeckSF+df.OpenPorchSF+df.EnclosedPorch+df.Threeseasonporch+df.ScreenPorch\n",
    "\n",
    "\n",
    "# Check your answer\n",
    "q_1.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5644dbda",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.542700Z",
     "iopub.status.busy": "2025-02-18T13:19:25.542310Z",
     "iopub.status.idle": "2025-02-18T13:19:25.553156Z",
     "shell.execute_reply": "2025-02-18T13:19:25.551869Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.018173,
     "end_time": "2025-02-18T13:19:25.554892",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.536719",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"1_Q1\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> Your code should look something like:\n",
       "```python\n",
       "X_1[\"LivLotRatio\"] = ____ / ____\n",
       "X_1[\"Spaciousness\"] = (____ + ____) / ____\n",
       "X_1[\"TotalOutsideSF\"] = ____ + ____ + ____ + ____ + ____\n",
       "```\n"
      ],
      "text/plain": [
       "Hint: Your code should look something like:\n",
       "```python\n",
       "X_1[\"LivLotRatio\"] = ____ / ____\n",
       "X_1[\"Spaciousness\"] = (____ + ____) / ____\n",
       "X_1[\"TotalOutsideSF\"] = ____ + ____ + ____ + ____ + ____\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"1_Q1\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "\n",
       "X_1[\"LivLotRatio\"] = df.GrLivArea / df.LotArea\n",
       "X_1[\"Spaciousness\"] = (df.FirstFlrSF + df.SecondFlrSF) / df.TotRmsAbvGrd\n",
       "X_1[\"TotalOutsideSF\"] = df.WoodDeckSF + df.OpenPorchSF + df.EnclosedPorch + df.Threeseasonporch + df.ScreenPorch\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "\n",
       "X_1[\"LivLotRatio\"] = df.GrLivArea / df.LotArea\n",
       "X_1[\"Spaciousness\"] = (df.FirstFlrSF + df.SecondFlrSF) / df.TotRmsAbvGrd\n",
       "X_1[\"TotalOutsideSF\"] = df.WoodDeckSF + df.OpenPorchSF + df.EnclosedPorch + df.Threeseasonporch + df.ScreenPorch\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "q_1.hint()\n",
    "q_1.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38645d64",
   "metadata": {
    "papermill": {
     "duration": 0.004962,
     "end_time": "2025-02-18T13:19:25.565119",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.560157",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "-------------------------------------------------------------------------------\n",
    "\n",
    "If you've discovered an interaction effect between a numeric feature and a categorical feature, you might want to model it explicitly using a one-hot encoding, like so:\n",
    "\n",
    "```\n",
    "# One-hot encode Categorical feature, adding a column prefix \"Cat\"\n",
    "X_new = pd.get_dummies(df.Categorical, prefix=\"Cat\")\n",
    "\n",
    "# Multiply row-by-row\n",
    "X_new = X_new.mul(df.Continuous, axis=0)\n",
    "\n",
    "# Join the new features to the feature set\n",
    "X = X.join(X_new)\n",
    "```\n",
    "\n",
    "# 2) Interaction with a Categorical\n",
    "\n",
    "We discovered an interaction between `BldgType` and `GrLivArea` in Exercise 2. Now create their interaction features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1aaf1edf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.576606Z",
     "iopub.status.busy": "2025-02-18T13:19:25.576207Z",
     "iopub.status.idle": "2025-02-18T13:19:25.594648Z",
     "shell.execute_reply": "2025-02-18T13:19:25.593484Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.026093,
     "end_time": "2025-02-18T13:19:25.596293",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.570200",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# YOUR CODE HERE\n",
    "# One-hot encode BldgType. Use `prefix=\"Bldg\"` in `get_dummies`\n",
    "X_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\") \n",
    "# Multiply\n",
    "X_2 = X_2.mul(df.GrLivArea, axis=0)\n",
    "\n",
    "\n",
    "# Check your answer\n",
    "q_2.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b54b368e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.608749Z",
     "iopub.status.busy": "2025-02-18T13:19:25.608343Z",
     "iopub.status.idle": "2025-02-18T13:19:25.618827Z",
     "shell.execute_reply": "2025-02-18T13:19:25.617927Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.018357,
     "end_time": "2025-02-18T13:19:25.620334",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.601977",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> Your code should look something like:\n",
       "```python\n",
       "X_2 = pd.get_dummies(____, prefix=\"Bldg\")\n",
       "X_2 = X_2.mul(____, axis=0)\n",
       "```\n"
      ],
      "text/plain": [
       "Hint: Your code should look something like:\n",
       "```python\n",
       "X_2 = pd.get_dummies(____, prefix=\"Bldg\")\n",
       "X_2 = X_2.mul(____, axis=0)\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"2_Q2\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "\n",
       "X_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\")\n",
       "X_2 = X_2.mul(df.GrLivArea, axis=0)\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "\n",
       "X_2 = pd.get_dummies(df.BldgType, prefix=\"Bldg\")\n",
       "X_2 = X_2.mul(df.GrLivArea, axis=0)\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "q_2.hint()\n",
    "q_2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c1b8559",
   "metadata": {
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.005538,
     "end_time": "2025-02-18T13:19:25.631853",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.626315",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Count Feature\n",
    "\n",
    "Let's try creating a feature that describes how many kinds of outdoor areas a dwelling has. Create a feature `PorchTypes` that counts how many of the following are greater than 0.0:\n",
    "\n",
    "```\n",
    "WoodDeckSF\n",
    "OpenPorchSF\n",
    "EnclosedPorch\n",
    "Threeseasonporch\n",
    "ScreenPorch\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d464108d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.644734Z",
     "iopub.status.busy": "2025-02-18T13:19:25.644315Z",
     "iopub.status.idle": "2025-02-18T13:19:25.661657Z",
     "shell.execute_reply": "2025-02-18T13:19:25.660463Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.025813,
     "end_time": "2025-02-18T13:19:25.663406",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.637593",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"3_Q3\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_3 = pd.DataFrame()\n",
    "\n",
    "# YOUR CODE HERE\n",
    "X_3[\"PorchTypes\"] = df[[\n",
    "    'WoodDeckSF',\n",
    "    'OpenPorchSF',\n",
    "    'EnclosedPorch',\n",
    "    'Threeseasonporch',\n",
    "    'ScreenPorch',\n",
    "]].gt(0.0).sum(axis=1)\n",
    "\n",
    "\n",
    "# Check your answer\n",
    "q_3.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0131ed07",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.677088Z",
     "iopub.status.busy": "2025-02-18T13:19:25.676752Z",
     "iopub.status.idle": "2025-02-18T13:19:25.687010Z",
     "shell.execute_reply": "2025-02-18T13:19:25.686039Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.019041,
     "end_time": "2025-02-18T13:19:25.688731",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.669690",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"3_Q3\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> Your code should look someting like:\n",
       "```python\n",
       "X_3 = pd.DataFrame()\n",
       "\n",
       "X_3[\"PorchTypes\"] = df[[\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "]].____.sum(axis=1)\n",
       "```\n"
      ],
      "text/plain": [
       "Hint: Your code should look someting like:\n",
       "```python\n",
       "X_3 = pd.DataFrame()\n",
       "\n",
       "X_3[\"PorchTypes\"] = df[[\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "    ____,\n",
       "]].____.sum(axis=1)\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"3_Q3\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "\n",
       "X_3 = pd.DataFrame()\n",
       "\n",
       "X_3[\"PorchTypes\"] = df[[\n",
       "    \"WoodDeckSF\",\n",
       "    \"OpenPorchSF\",\n",
       "    \"EnclosedPorch\",\n",
       "    \"Threeseasonporch\",\n",
       "    \"ScreenPorch\",\n",
       "]].gt(0.0).sum(axis=1)\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "\n",
       "X_3 = pd.DataFrame()\n",
       "\n",
       "X_3[\"PorchTypes\"] = df[[\n",
       "    \"WoodDeckSF\",\n",
       "    \"OpenPorchSF\",\n",
       "    \"EnclosedPorch\",\n",
       "    \"Threeseasonporch\",\n",
       "    \"ScreenPorch\",\n",
       "]].gt(0.0).sum(axis=1)\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "q_3.hint()\n",
    "q_3.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2876cfe",
   "metadata": {
    "papermill": {
     "duration": 0.006091,
     "end_time": "2025-02-18T13:19:25.701580",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.695489",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4) Break Down a Categorical Feature\n",
    "\n",
    "`MSSubClass` describes the type of a dwelling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6c6140dd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.715621Z",
     "iopub.status.busy": "2025-02-18T13:19:25.715221Z",
     "iopub.status.idle": "2025-02-18T13:19:25.722042Z",
     "shell.execute_reply": "2025-02-18T13:19:25.720928Z"
    },
    "papermill": {
     "duration": 0.01627,
     "end_time": "2025-02-18T13:19:25.724104",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.707834",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['One_Story_1946_and_Newer_All_Styles', 'Two_Story_1946_and_Newer',\n",
       "       'One_Story_PUD_1946_and_Newer',\n",
       "       'One_and_Half_Story_Finished_All_Ages', 'Split_Foyer',\n",
       "       'Two_Story_PUD_1946_and_Newer', 'Split_or_Multilevel',\n",
       "       'One_Story_1945_and_Older', 'Duplex_All_Styles_and_Ages',\n",
       "       'Two_Family_conversion_All_Styles_and_Ages',\n",
       "       'One_and_Half_Story_Unfinished_All_Ages',\n",
       "       'Two_Story_1945_and_Older', 'Two_and_Half_Story_All_Ages',\n",
       "       'One_Story_with_Finished_Attic_All_Ages',\n",
       "       'PUD_Multilevel_Split_Level_Foyer',\n",
       "       'One_and_Half_Story_PUD_All_Ages'], dtype=object)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.MSSubClass.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb89170b",
   "metadata": {
    "papermill": {
     "duration": 0.00619,
     "end_time": "2025-02-18T13:19:25.737272",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.731082",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You can see that there is a more general categorization described (roughly) by the first word of each category. Create a feature containing only these first words by splitting `MSSubClass` at the first underscore `_`. (Hint: In the `split` method use an argument `n=1`.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c0473dbd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.751811Z",
     "iopub.status.busy": "2025-02-18T13:19:25.751394Z",
     "iopub.status.idle": "2025-02-18T13:19:25.769787Z",
     "shell.execute_reply": "2025-02-18T13:19:25.768766Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.027485,
     "end_time": "2025-02-18T13:19:25.771380",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.743895",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4_Q4\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_4 = pd.DataFrame()\n",
    "\n",
    "# YOUR CODE HERE\n",
    "X_4[\"MSClass\"] = df.MSSubClass.str.split(\"_\",n=1,expand=True)[0]\n",
    "\n",
    "# Check your answer\n",
    "q_4.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "27443b67",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.787145Z",
     "iopub.status.busy": "2025-02-18T13:19:25.786794Z",
     "iopub.status.idle": "2025-02-18T13:19:25.797580Z",
     "shell.execute_reply": "2025-02-18T13:19:25.796555Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.02053,
     "end_time": "2025-02-18T13:19:25.799178",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.778648",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"4_Q4\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> Your code should look something like:\n",
       "```python\n",
       "X_4 = pd.DataFrame()\n",
       "\n",
       "X_4[\"MSClass\"] = df.____.str.____(____, n=1, expand=True)[____]\n",
       "```\n"
      ],
      "text/plain": [
       "Hint: Your code should look something like:\n",
       "```python\n",
       "X_4 = pd.DataFrame()\n",
       "\n",
       "X_4[\"MSClass\"] = df.____.str.____(____, n=1, expand=True)[____]\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"4_Q4\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "\n",
       "X_4 = pd.DataFrame()\n",
       "\n",
       "X_4[\"MSClass\"] = df.MSSubClass.str.split(\"_\", n=1, expand=True)[0]\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "\n",
       "X_4 = pd.DataFrame()\n",
       "\n",
       "X_4[\"MSClass\"] = df.MSSubClass.str.split(\"_\", n=1, expand=True)[0]\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "q_4.hint()\n",
    "q_4.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b54fa2cc",
   "metadata": {
    "papermill": {
     "duration": 0.007127,
     "end_time": "2025-02-18T13:19:25.814200",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.807073",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5) Use a Grouped Transform\n",
    "\n",
    "The value of a home often depends on how it compares to typical homes in its neighborhood. Create a feature `MedNhbdArea` that describes the *median* of `GrLivArea` grouped on `Neighborhood`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f3c6d020",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.830269Z",
     "iopub.status.busy": "2025-02-18T13:19:25.829878Z",
     "iopub.status.idle": "2025-02-18T13:19:25.847745Z",
     "shell.execute_reply": "2025-02-18T13:19:25.846733Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.028006,
     "end_time": "2025-02-18T13:19:25.849476",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.821470",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.2, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"5_Q5\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc33\">Correct</span>"
      ],
      "text/plain": [
       "Correct"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_5 = pd.DataFrame()\n",
    "\n",
    "# YOUR CODE HERE\n",
    "X_5[\"MedNhbdArea\"] = df.groupby(\"Neighborhood\")[\"GrLivArea\"].transform(\"median\")\n",
    "\n",
    "# Check your answer\n",
    "q_5.check()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a1476117",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.866054Z",
     "iopub.status.busy": "2025-02-18T13:19:25.865714Z",
     "iopub.status.idle": "2025-02-18T13:19:25.875929Z",
     "shell.execute_reply": "2025-02-18T13:19:25.874867Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.020644,
     "end_time": "2025-02-18T13:19:25.877904",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.857260",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 2, \"questionType\": 2, \"questionId\": \"5_Q5\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#3366cc\">Hint:</span> Your code should look something like:\n",
       "```python\n",
       "X_5 = pd.DataFrame()\n",
       "\n",
       "X_5[\"MedNhbdArea\"] = df.____(\"Neighborhood\")[\"____\"].transform(____)\n",
       "```\n"
      ],
      "text/plain": [
       "Hint: Your code should look something like:\n",
       "```python\n",
       "X_5 = pd.DataFrame()\n",
       "\n",
       "X_5[\"MedNhbdArea\"] = df.____(\"Neighborhood\")[\"____\"].transform(____)\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"interactionType\": 3, \"questionType\": 2, \"questionId\": \"5_Q5\", \"learnToolsVersion\": \"0.3.4\", \"valueTowardsCompletion\": 0.0, \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\", \"outcomeType\": 4}}, \"*\")"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "<span style=\"color:#33cc99\">Solution:</span> \n",
       "```python\n",
       "\n",
       "X_5 = pd.DataFrame()\n",
       "\n",
       "X_5[\"MedNhbdArea\"] = df.groupby(\"Neighborhood\")[\"GrLivArea\"].transform(\"median\")\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "Solution: \n",
       "```python\n",
       "\n",
       "X_5 = pd.DataFrame()\n",
       "\n",
       "X_5[\"MedNhbdArea\"] = df.groupby(\"Neighborhood\")[\"GrLivArea\"].transform(\"median\")\n",
       "\n",
       "```"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "q_5.hint()\n",
    "q_5.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2740410d",
   "metadata": {
    "papermill": {
     "duration": 0.007521,
     "end_time": "2025-02-18T13:19:25.893498",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.885977",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now you've made your first new feature set! If you like, you can run the cell below to score the model with all of your new features added:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6c943684",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-18T13:19:25.910671Z",
     "iopub.status.busy": "2025-02-18T13:19:25.910261Z",
     "iopub.status.idle": "2025-02-18T13:19:28.504027Z",
     "shell.execute_reply": "2025-02-18T13:19:28.503053Z"
    },
    "papermill": {
     "duration": 2.605589,
     "end_time": "2025-02-18T13:19:28.506906",
     "exception": false,
     "start_time": "2025-02-18T13:19:25.901317",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.13954039790897127"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = X.join([X_1, X_2, X_3, X_4, X_5])\n",
    "score_dataset(X_new, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e681f025",
   "metadata": {
    "papermill": {
     "duration": 0.008308,
     "end_time": "2025-02-18T13:19:28.527045",
     "exception": false,
     "start_time": "2025-02-18T13:19:28.518737",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Keep Going #\n",
    "\n",
    "[**Untangle spatial relationships**](https://www.kaggle.com/ryanholbrook/clustering-with-k-means) by adding cluster labels to your dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9c94605",
   "metadata": {
    "papermill": {
     "duration": 0.008991,
     "end_time": "2025-02-18T13:19:28.544681",
     "exception": false,
     "start_time": "2025-02-18T13:19:28.535690",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/feature-engineering/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb"
  },
  "kaggle": {
   "accelerator": "none",
   "dataSources": [
    {
     "datasetId": 933090,
     "sourceId": 1828856,
     "sourceType": "datasetVersion"
    }
   ],
   "isGpuEnabled": false,
   "isInternetEnabled": false,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 9.685732,
   "end_time": "2025-02-18T13:19:29.278901",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2025-02-18T13:19:19.593169",
   "version": "2.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
